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standardized Mass Scraeninq of "At Risk" Students; 

A Multiple Gating Model 

Abstract: 

Reliability and validity datd on a procedure for 
identification of students "at risk" for exhibiting 
behavior disorders is presented. Systematic Screening for 
Behavior Disorders (SSBD) relies upon teacher judgment and 
noriaative criteria in three interrelated stages that cross 
validate the results of each other. Stage One has teachers 
rank students on both "externalizing" and "internalizing" 
behavioral dimensions. Stage Two involves teacher 
completed rating measures on adaptive and maladaptive 
behavior while Stage Three includes observation in 
classroom and free play settings which are used to confirm 
other measures. Data is presented on a district wide trial 
involving 158 classroom teachers in 16 schools. 
Observation data on 301 elementary grade students confirm 
the accuracy of this ::ost effective procedure in 
identifying children exhibiting behavior disorders. 
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STANDARDIZED MASS SCREENING OF AT RISK STUDENTS: 
A MULTIPLE GATING MODEL 

Surveys of school district practices in the 
certification of students as behavior disordered (BD) or 
severely emotionally disturbed (SED) consistently yield 
results that suggest high levels of dissatisfaction with 
current assessment procedures which are often conducted by 
school psychologists who perceive themselves as 
inadequately trained for this task (Gresham, in press). 
Reflecting current levels of issatisf action with such 
practices, the Haring et al (1985) study of BD assessment 
and service delivery practices in Washington State, 
identified two areas of critical need: 1) the need for 
agreement with regard to identification and eligibility 
criterion for students with severe behavior disabilities, 
and 2) the need to develop and implement a comprehensive 
assessment procedure for this population. Additionally, 
research literature has documented the subjective and 
arbitrary nature of assessment procedures used with BD 
children (Gerber and Semmel, 1984) . 

In response to need for a standardized assessment 
procedure, Drs. Walker and Severson developed a screening 
instrument entitled. Systematic Screening for Behavior 
Disorders (SSBD) (Walker, Severson & Haring. 1985). The 
assessment procedure provides for cost-effective, mass 
screening of all children who are enrolled in regular 
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classrooms and links: a) definition criteria, b) screening 
and assessment procedures, and c) normative based 
eligibility decision making into one self-contained system. 
This model relies heavily upon structured teacher judgment 
of child behavioral characteristics in the first two 
assessment stages and uses normatively referenced 
observational data to provide independent in vivo 
assessments of the child's functioning within instructional 
and free play settings in Stage Three. The results of 
assessment and decision making in initial stages are cross 
validated by increasingly more intensive assessments in 
subsequent stages or "gates." This multiple gating 
assessment model provides a cost effective system for 
screening and identification of elementary grade children 
"at risk" for exhibiting behavior disorders (Figure 1). 

Stage one of the screening procedures asks the 
classroom teacher to rank order students on two behavioral 
dimensions. The first dimension is on externalizing 
behavior problems which are behaviors directed outwardly by 
the child toward the external social environment and 
usually involve behavioral excesses. The teacher also 
ranks students on internalizing behavioral problems that 
are directed inwa. =lly and frequently involve behavioral 
deficits as welj as patterns of social avoidance. This 
bipolar classification system has been found consistently 
in factor analytic studies of behavioral disorders (Ross, 
1980). The structured teacher judgments involved in 
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externalizing and internalizing behavior problems are 
ranked relative to classroom peers. 

Stage Two procedures involve the completion of 
behavior rating scales and critical behavioral indices for 
the top three ranked children on both behavioral dimensions 
from Staga One. Students at Stage Two are compared with 
behavioral norms for these rating scales and if they exceed 
cut-off scores, the child is assessed using Stage Three 
direct observation procedures. 

Stage Three involves two areas of behavioral 
observation: 1) within class Academic Engaged Time (AET) , 
and 2) observation of Peer Social Behavior (PSB) in a free 
play setting. 

The authors of the 3SBD screening instrument have 
conducted pilot research and field test trials to establish 
reliability and discriminative validity for this 
instrument. Inter-rater reliability coefficients for 
externalizing behavior dimensions range from .89 to .94 and 
internalizing behavior dimensions range from .73 to .88. 
Test/retest reliability cctfficients on teacher rankings 
with a one month time interval were .76 for externalizers 
and .74 for internalizers . 

Coefficient alpha was completed for cumulative 
frequency indices in Stage Two instruments. For adaptive 
behavior the coefficient alpha was .85 and .88 in two 
cohorts of 108 pupils each. For the maladaptive behavior 
the figures were. 82 and .87. Item analysis on these 
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scales revealed consistent positive items to total score 
correlations. The classroom and peer interaction 
observation codes also appear very reliable. Inter- 
observer reliability of the AET classroom code was .96 
(range .84 to 1.0) on 39 reliability checks, while the PSB 
code had a mean reliability of .88 (range .80 to 1.0) in 63 
reliability checks (Table 2). 

The PSB observation data clearly shows that students 
identified by the classroom teacher as exhibiting 
externalizing or internalizing behavior differ 
significantly both from non-identified "normals" or control 
subjects and each other. Observation data shown in Tables 
2, 3, and 4 reveals significant differences in observed 
behavior for all three groups (p<.01). The observers were 
blind to the classification done by teachers. The results 
of the observations indicate that the behavior patterns 
identified by teachers in the Stage One ranking procedures 
are confirmed by independent observers. Students 
classified as intemalizers , who are most often overlooked 
by referral sources, exhibit significantly more Alone 
behavior. Parallel Pla y, and are less Socially Engaged . 
Children classified as externalizers exhibit high levels of 
Participation , but also high levels of Negative Social 
Interaction . 

As a further test of the ability of the SSBD to screen 
for at risk students the data from Stages Two and Three was 
subjected to a discriminant function analysis. This 
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analysis (shown in Table 5 shows a high level of 
classification efficiency with 85% of the cases being 
correctly classified. The eight variables that enter into 
the discriminant function are shown in Table 6 with the 
Wilk's Lamda for each variable. Note that Adaptive 
Behavior and Maladaptive Behavior as rated by teachers on 
Stage Two instruments were the first two factors. 
Playground observation (PSB) variables that contributed 
most to the prediction of Stage One classification were 
Parallel Play, Social Involvement, Negative Interaction, 
and Social Engagement. A review of Table 2 ^combined sex 
groups) indicates the specific direction of each of these 
variables. 

The results of Stage Two behavior rating scales 
confirm the teacher discriminations made on Stage One. The 
results shown in TABLE 7 show significant differences in 
the three groups for all three measures at Stage Two. 
While these measures are also completed uy the teacher and 
one would expect correspondence with their rankings at 
Stage One, the instruments at Stage Two are behaviorally 
specific and involve a measure of frequency. Results of 
the Critical Events Checklist indicate that externalizers 
display twice as many problematic behaviors when compared 
to controls. All group differences and sex differences are 
in the expected direction and of significant magnitude 
(p<.01) . 
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Table 8 shows the results of the classroom 
observations completed on 298 subjects. There were 
significant differences in Academic Engaged Time (AET) 
between non-identified (control) subjects and both 
internalizers and externalizers . Previous research has 
also shown significant differences between the 
internalizing and externalizing groups (Walker, et al 
1984) . The non-significant results for this sample may be 
due to a large number of observations being completed on 
lower ranked students (on Stage One) because parents of 
children ranked first by the teacher refused to provide 
positive consent to observe their child. 

Our research group has now completed the third year of 
developmental research on the screening procedures. The 
instrument has proven to be highly reliable, and cost 
effective for identifying students that are exhibiting 
behaviors that put the child "at risk" for subsequent 
behavior disorders. The current data is from a district 
wide assessment of the SSBD systeji. This study involved 
158 teachers from 16 schools completing Stage one and Stage 
Two instruments. Project staff observed 97 classrooms and 
301 children from grades one through five. We now have the 
beginning of a norm sample of children with which to 
compare teacher ratings and observed behavior in both 
classroom and playground settings. This federally funded 
project will continue to assess the use of this screening 
procedure with field implementation trials in several 
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school districts in the 1987-88 school year. The project 
is also collecting "student record searches" to provide an 
additional validation of the screening procedure. This 
assessment of academic performance, standardized 
achievement scores, referrals and discipline contacts will 
be analyzed to determine the differences in these measures 
for students identified by the screening procedure. 
Follow-up longitudinal tracking of highly ranked students 
will also determine the stability of these rankings across 
classroom environments . 
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TABLE 1 

RELIABILITY OF OBSERVA'^ jU INSTRUMENTS (SSBD) 



Total Observations n = 772 

Playground 386 
Classroom 386 



Reliability Checks n = 1« 2 13.4% (102/772) 

Playground 63 16.3% (63/386) 

Classroom 39 10.5% (39/386) 



Reliability Coefficients X of Playground = .88 Range = .80 to 1.00 

X of Classroom = .96 Range = .84 to 1.00 
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TABLE 2 

SSBD PEER SOCIAL BEHAVIOR OBSERVATIONS 
MALES & FEMALES 



PSB Dependent Variable 

1 Socially Engaged 
(SE) 



Externalizers 
(n=73) 

X 

SD 

3 

30.1 
16.8 



Internalizers 
(n=76) 

X 

SD 

2 

27 . 4 
13 . 1 



Controls 
(n=l52) 

X 

SD 

35. 1 
15. 3 



Socially Involved 
(SI) 



32.9 
15.6 



1,3 



43.9 
15. 3 



40.9 
16.2 



Participation 
(P) 



22 . t: 

28.7 



7.4 
16.1 



14.2 
22. 6 



Parallel Play 
(PLP) 



5.8 
7 . 5 



10.7 
10. 6 



4.9 
7.5 



5 Alone 
(A) 

6 No Codeable Response 



Social Interaction 
(SE + SI) 



6.1 
8.3 

1 . 5 
1.7 

64 .7 
26.4 



8 . 6 
10.0 

1.7 
3 . 6 

71.5 
19.5 



2 , 3 



3.5 
5.5 

1.3 
2.0 

76.0 
22.2 



Positive Interaction 
(SE + SI +) 



58 . 8 
25.3 



69.9 
19.6 



75.2 
21.9 



Negative Social Interaction 5.4 
(SE - + SI -) 7.5 



1,3 



1.8 
4 . 2 



1.4 

2 . 9 



10 Total Positive Behavior 
(SE + SI + P) (+) 

11 Total Negative Behavior 



80.5 
17 . 2 

6.0 
7.9 



1,3 



1 = Externalizers vs. Controls p<.05 

2 = Internalizers vs. Controls p<.05 

3 = Externalizers vs. Internalizers p<.05 



77. 2 
18.2 

1.8 
4 . 2 



88 . 6 
11. 5 

1.9 
5.6 
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TABLE 3 



SSBD PEER SOCIAL BEHAVIOR OBSERVATIONS 

MALES ONLY 



Externalizers 


Internal izers 


Controls 




(n=73) 




(rt = fi R ^ 




X 


X 


X 


rSB Dependent Variable 


SD 


SD 


o u 


1 Socially Engaged* 


29. 9 


26.3 


31.5 


(SE) 


16.5 


11 . 1 


15 . 2 


2 Socially Involved 


32.6 


45 . 6 


38.7 


(SI) 


16.1 


15 . 7 




3 Participation* 


24.6 


8.0 


20.2 


(P) 


29.6 


16.9 




4 Parallel Play* 


5 . 1 


10.5 


4.3 


(PLP) 


5.8 


9 1 


D . ^ 


5 Alone 


6 . 2 


7 . 9 


4.1 


(A) 


8.5 


8 . 6 


O.J. 


6 No Codeable Response 


1.6 


1 . 1 


1.2 




1.9 


1.9 


1.7 


7 Social Interaction* 


63. 6 


72 . 4 


69.7 


(SE + SI) 


26.6 


18. 1 


25.7 


8 Positive Interaction* 


57. 6 


69. 8 


68. 7 


(SE + SI +) 


25. 1 


18.5 


25.3 


9 Negative Social Interaction 


5.4 


2.9 


1.8 


(SE - + SI -) 


7.6 


6.5 


3.4 


10 Total Positive Behavior 


81.1 


77. 7 


87 . 9 


(SE + SI + P) (+) 


15 . 6 


15 . 4 


11.5 


11 Total Negative Behavior 


6.0 


2.7 


2 . 8 




7.9 


5.4 


7.5 


*sex differences p<.05 
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TABLE 4 

SSBD PEER SOCIAL BEHAVIOR OBSERVATIONS 
FEMALES ONLY 



Externalizers Inrernalizers Controls 
(n=8) (n=37) (n=76) 



PSB Dependent Variable 

1 Socially Engaged* 
(SE) 

2 Socially Involved 
(SI) 

3 Participation* 
(P) 

4 Parallel Play* 
(PLP) 

5 Alone 
(A) 

6 No Codeable Response 

7 Social Interaction* 
(SE + SI) 

8 Positive Interaction* 
(SE + SI +) 

9 Negative Social Interaction 
(SE - + SI -) 

10 Total Positive Behavior 
(SE + SI + p) (+) 

11 Total Negative Behavior 
*sex differences p<.05 



X 


X 


Y 


SD 


SD 


SD 


38.8 


28.5 


o • / 


18 . 0 


14.9 


14. 6 


35.3 


42.1 




12 .2 


14.8 


12.8 


6 . 7 


6 . 8 


o • «^ 


11.2 


15. 5 


15.3 


11 . 5 


10.9 




15.2 


12 . 1 


8.5 


5 . 3 


9 . 3 




6.9 


11 . 8 


4 8 


1.0 


2.4 


1.3 


1.6 


4.6 


2.3 


74 .3 


70.6 


82.3 


24 . 0 


21.1 


15.8 


68 . 6 


70.0 


81.8 


26.1 


20.9 


15.5 


5.6 


.7 


1.0 


7.7 


1.8 


2.3 


75.7 


76.7 


89.3 


28. 4 


20.9 


11.5 


5.6 


. 8 


1.0 


7.7 


1.9 


2.3 
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TABLE 5 

DISCRIMINANT FUNCTION ANALYSIS 



Number Predicted Group Membership 

Actual Group of Cases Non-Identified Externalizer Internali 



Non-Identified 150 



Externalizer 



Internalizer 



6y 



7o 



142 
94 . 7% 

4 

5.8% 

13 
17.3% 



2 

1.3% 

56 
81.2% 

11 
14 .7% 



6 

4.0% 
9 

13.0% 

51 
68 . 0% 



Percent of "grouped" cases correctly c.assified: 84.69% 
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TABLE 6 



ORDER OF VARIABLES 


IN SSBD 


DISCRIMINANT 


FUNCTION 




Vars 


Wilks ' 




Step Entered 


in 


Lambda 


Signif j canoe 


1 Adaptive 


1 


. 37379 


. 000 


2 Maladptive 


2 


. 27463 


. 000 


3 Parallel Play 


3 


. 25754 


. 000 


4 Critical Events 


4 


.24364 


. 000 


5 Social Involvement 


5 


. 23258 


. 000 


6 Negative Interaction 


6 


. 22124 


. 000 


7 Social Engagement 


7 


. 21690 


. 000 


8 Academic Engaged Time 


8 


. 21381 


. 000 
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TABLE 7 



SSBD STAGE TWO INSTRUMENTS 



Externalizers Internalizers Controls 
Males Females Males Females Males Females 
(n=247) (n=56) (N=142) (n=150) (n=116) (n=109) 



Critical Events Index 



X 
SD 



3.2 
2.6 



3 . 1 
2.9 



1.9 
1.8 



2.2 
1.9 



2.1 
2.4 



1,2,3 
1.6 
2.1 



Combined Frequency Index 
Adaptive Behavior 



12 3 

X 3.05 3.08 3.59 3.84 3.54 4.o' ' 

SD .64 .65 .66 .66 .84 .80 



Maladaptive Behavior 

SD .68 .80 .55 .49 .88 .72 



X 2.77 2.76 1.73 1.66 2.13 1.71^'^ 



1 = Controls vs. Externalizers p<.01 

2 = Controls vs. Internalizers p<.01 

3 = Externalizers vs. Internalizers p<.01 
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TABLE 8 



SSBD CLASSROOM OBSERVATIONS 



Externalizers Internalizers Controls 

70) (n = 75) (n = 153) 

Academic Engaged X 65 X731 X''?? 

Time (AET) SD 18.1 SD 15.' 2 3D 13 ! 7 



Controls vs. Externalizers p<.01 
Controls vs. Internalizers p<.03 
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